Tileable non-planar structured light patterns for wide field-of-view depth sensing

ABSTRACT

A head-mounted display (HMD) system includes a projector assembly configured to emit a structured light (SL) pattern onto one or more objects in a local area, the projected SL pattern comprises at least a first SL pattern having a first field of view (FOV) corresponding to a first tileable boundary, and a second SL pattern having a second FOV corresponding to a second tileable boundary. The first and second SL patterns are projected such that the first and second tileable boundaries share at least one edge and collectively define a non-planar surface. A detector assembly is configured to capture one or more images of the one or more objects in the local area illuminated by the tiled SL pattern, such that a location of the HMD may be determined using the one or more captured images.

BACKGROUND

The present disclosure generally relates to design of a structured lightpattern, and specifically relates to generating a tileable structuredlight projection for wide field-of-view (FOV) depth sensing.

Structured light (SL) patterns may be used to determine a position ororientation of a head-mounted display (HMD) worn by a user within alocal area. For example, a projector system may project an SL patternover objects within a local area. Imaging devices on the HMD captureimages of the local area, including at least a portion of the projectedSL patterns. The positions of the SL patterns within the captured imagesis used to determine a depth of the objects relative to the HMD and/or aposition of the HMD within the local area.

The SL pattern may be projected using a diffractive optical element(DOE). However, it may be difficult to use a single DOE to project an SLpattern over a wide field-of-view (FOV). For example, using a singlewide FOV DOE may lead to large zero-order values compared to otherdiffraction orders, which may lead to issues in laser safety complianceand algorithm performance.

SUMMARY

In some embodiments, a head-mounted display (HMD) is provided. The HMDsystem includes a projector assembly configured to emit a structuredlight (SL) pattern onto one or more objects in a local area, theprojected SL pattern comprises at least a first SL pattern having afirst field of view (FOV) corresponding to a first tileable boundary,and a second SL pattern having a second FOV corresponding to a secondtileable boundary. The first and second SL patterns are projected suchthat the first and second tileable boundaries share at least one edgeand collectively define a non-planar surface. A detector assembly isconfigured to capture one or more images of the one or more objects inthe local area illuminated by the tiled SL pattern, such that a locationof the HMD may be determined using the one or more captured images.

In some embodiments, the projector assembly projects the first andsecond SL patterns by emitting light that is diffracted by first andsecond respective augmented diffractive optical elements (ADOEs). AnADOE is a diffractive optical element that is designed to diffract lightinto a SL pattern projection that has a field of view (FOV)corresponding to a respective tileable boundary, and prevents projectionof portions of the SL pattern that would otherwise lie outside therespective tileable boundary. In some embodiments, an ADOE is designedby, e.g., making a design pattern such that light diffracted from theADOE is within a FOV bounded by the respective tileable boundary, andthen proceeding with a normal lithography process to form a diffractiveelement.

In some embodiments, the projector assembly projects multiple SLpatterns each corresponding to a respective tileable boundary arrangedin a tessellated manner. This allows for the projector assembly toproject SL patterns over a wider FOV within the local area, whileminimizing gaps or overlaps between different SL patterns. In addition,by projecting the SL patterns such that the tileable boundaries of theSL patterns define a non-planar surface, a more uniform SL pattern maypotentially be projected onto objects in the local area.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a head-mounted display system, in accordance with anembodiment.

FIG. 2 is a wire diagram of a head-mounted display, in accordance withan embodiment.

FIG. 3 is a block diagram of a projector system including multipleprojection assemblies, in accordance with an embodiment.

FIG. 4 illustrates a diagram of a projector system projecting tileablelight patterns, in accordance with some embodiments.

FIG. 5 illustrates another diagram of a projector projecting tileablenon-coplanar SL patterns, in accordance with some embodiments.

FIG. 6 illustrates a projector projecting a SL pattern associated with acurved surface, in accordance with some embodiments.

FIG. 7 illustrates a flowchart of a process for projecting SL patternsin accordance with some embodiments.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

Structured light (SL) illumination (also referred to as SL patternprojection) with a wide field-of-view (FOV) is crucial for achievingwide FOV depth sensing of target objects. Disclosed embodiments includea method and apparatus for achieving an efficient wide FOV illuminationby tiling multiple SL pattern projections in a non-planar arrangement.Each SL pattern projection is associated with a high-performingrectangular FOV pattern projection (or, alternatively, hexagonal,square, etc.) obtained based on one or more diffractive opticalelements.

Disclosed embodiments include a head-mounted display (HMD) system havinga depth camera assembly (DCA) for determining depth information of oneor more objects in a local area. The DCA includes one or more SLsources, a detector, and a controller. Each SL source integrated intothe DCA emits a SL pattern projection into the local area. The SL sourceincludes an augmented diffractive optical element (ADOE) illuminatedwith a plurality of light emitters of the SL source. An ADOE is adiffractive optical element that is designed to diffract light into a SLpattern projection that has a FOV corresponding to a tileable boundary(e.g., rectangular, hexagonal, square, etc.). The ADOE is designed toprevent projection of portions of the SL pattern projection that wouldotherwise lie outside the tileable boundary, which also saves energy ofthe SL source. Each ADOE in the DCA generates a SL pattern projectionthat is projected into the local area in a tileable manner. One or moreADOEs in the DCA are designed such that they generate a tiled lightprojection which can be easily tiled with other similar projections. Theangular spacing between features in each ADOE is constant—and theygenerally have a barrel shaped boundary that cuts off what wouldotherwise result in the light projection having a shape that is noteasily tileable (e.g., pincushion). The detector of the DCA captures oneor more images of the local area illuminated with tiled lightprojections. The controller of the DCA determines depth information forobjects in the local area using the one or more images.

In some embodiments, at least a portion of the DCA is integrated into ahead-mounted display (HMD) that captures data describing depthinformation in a local area surrounding some or all of the HMD. The HMDmay be part of, e.g., a virtual reality (VR) system, an augmentedreality (AR) system, a mixed reality (MR) system, or some combinationthereof. The HMD may further include an electronic display and anoptical assembly. The electronic display is configured to emit imagelight. The optical assembly is configured to direct the image light toan exit pupil of the HMD corresponding to a location of a user's eye,the image light comprising the depth information of the one or moreobjects in the local area determined by the DCA.

In some embodiments, the DCA with the ADOE is integrated into a stationseparate from a HMD. In one embodiment, the station is a consoleinterfaced through a wired connection to a HMD. In an alternateembodiment, the station is a base station that wirelessly communicateswith the HMD. The DCA of the station captures data describing depthinformation in an environment surrounding some or all of the stationincluding a user wearing the HMD. The station may provide the depthinformation to the HMD, which is presented as content to the user.

System Overview

FIG. 1 is a block diagram of an HMD system 100, in accordance with anembodiment. The HMD system 100 may operate in a VR system environment,an AR system environment, a MR system environment, or some combinationthereof.

The HMD system 100 shown by FIG. 1 comprises a head-mounted display(HMD) 105 that includes a console 110, and an imaging device 135, aprojector system 136, and an input interface 140 that are each coupledto the console 110. While FIG. 1 shows an example system 100 includingone HMD 105, one projector system 136, and one input interface 140, inother embodiments any number of these components may be included in thesystem 100. For example, there may be multiple projector systems 136projecting a plurality of light patterns in a local area including theprojector system 136, which the HMD 105 and/or the input interface 140use to orient themselves within a virtual mapping of the local area. Inthe preceding example, each HMD 105, input interface 140, projectorsystem 136, and imaging device 135 communicates with the console 110. Inalternative configurations, different and/or additional components maybe included in the system 100.

The projector system 136 includes one or more projectors that generateand project one or more SL patterns throughout a local area thatincludes the projector system 136. In some embodiments, the projectorsystem 136 includes one or more light sources that emit coherent lightat specific bands (e.g., a range of wavelengths of light). Example bandsof light emitted by one or more light sources in the projector system136 include a visible band (˜380 nm to 750 nm), an infrared (IR) band(˜750 nm to 1 mm), an ultraviolet band (10 nm to 380 nm), anotherportion of the electromagnetic spectrum, or some combination thereof.For example, a light source in the projector system 136 can be a laserproducing light in the IR band. In some embodiments, a light source ofthe projector system 136 may be composed of a plurality of laser-typelight emitters on a single substrate configured to simultaneously emit aplurality of light beams to form a SL pattern projection.

To generate SL patterns, the projector system 136 comprises one or morediffractive optical elements that are illuminated by the one or morelight sources in the projector system 136. The generated light patternsare then projected into the local area by the projector system 136. Insome embodiments, the diffractive optical elements comprise one or moreADOEs placed in front of the light sources to form a respectiveprojected SL pattern. As discussed above, an ADOE is a diffractiveoptical element that is modified to diffract light emitted from anassociated light source into a SL pattern projection that has a FOVcorresponding to a tileable boundary of a suitable shape (e.g.,rectangular, hexagonal, square, etc.). Each ADOE is designed to preventprojection of portions of the SL pattern projection that would otherwiselie outside its associated tileable boundary. In some embodiments, apattern of the ADOE is designed with a pattern mask to preventprojection of portions of the SL pattern projection that would lieoutside the desired tileable boundary.

An SL pattern projection with FOV corresponding to a respective tileableboundary can be combined with at least one other SL pattern projectiongenerated by the projector system 136 to form a tiled light projectionthat illuminates the local area with a wide FOV. The tiled lightprojection represents a SL pattern composed of multiple non-overlappingSL pattern projections that illuminates one or more objects in the localarea achieving a wide FOV while mitigating distortions. In someembodiments, the multiple non-overlapping projected SL patterns aretessellated to collectively define a planar surface. In otherembodiments, the multiple non-overlapping projected SL patterns aretessellated to collectively define a non-planar surface.

An SL pattern as used herein may refer to a pattern or configuration oflight beams that may be projected onto one or more objects in a localarea surrounding the projector system 136. In some embodiments, aprojected SL pattern comprises different light patterns that areassociated with different locations in a virtual mapping of the localarea, the local area corresponding to a real world environment ontowhich the projector system 136 projects the SL patterns. For example, alocal area may correspond to an interior of a room enclosing theprojector system 136 that projects SL patterns onto one or more surfaceswithin the room (e.g., the walls and ceiling of the room, objects withinthe room, and/or the like).

In some embodiments, each SL pattern has a unique locationconfiguration, which describes a spatial configuration of light beams ofthe SL pattern and a reflectance type of the SL pattern. The spatialconfiguration of an SL pattern describes a number and an arrangement ofilluminated regions within the SL pattern, while the reflectance typespecifies a band of light (e.g., a range of wavelengths of light) usedto generate the SL pattern. In some embodiments, the projector system136 projects a plurality of SL patterns, of which no two SL patternshave the same unique location configuration. For example, each SLpattern may have a different spatial configuration, but have a commonreflectance type. Alternatively, multiple SL patterns may have the samespatial configuration but have different reflectance types.

The projector system 136 generates the SL patterns based upon receivedemission instructions that control one or more operating parameters ofthe projector system 136. For example, the emission instructions mayindicate operating parameters corresponding to, e.g., wavelength,modulation, pulse rate, pulse duration, amplitude, ADOE selection, someother operation of the projector system 136, or some combinationthereof. In some embodiments, emission instructions for the projectorsystem 136 are generated by and received from a controller at a console(e.g., the controller 150 at the console 110).

The HMD 105 may act as a VR, AR, and/or a MR HMD. An MR and/or AR HMDaugments views of a physical, real-world environment withcomputer-generated elements (e.g., images, video, sound, etc.). The HMD105 presents content to a user. Example content includes images, video,audio, or some combination thereof. Audio content may be presented via aseparate device (e.g., speakers and/or headphones) external to the HMD105 that receives audio information from the HMD 105, the console 110,or both. The HMD 105 includes an electronic display 115, an optics block118, one or more position sensors 125, an inertial measurement unit(IMU) 130, an imaging device 135, and a tracking module 160. Theelectronic display 115 displays images to the user in accordance withdata received from the console 110. In various embodiments, theelectronic display 115 may comprise a single electronic display ormultiple electronic displays (e.g., a display for each eye of a user).Examples of the electronic display 115 include: a liquid crystal display(LCD), an organic light emitting diode (OLED) display, an active-matrixorganic light-emitting diode display (AMOLED), some other display, orsome combination thereof.

The optics block 118 magnifies received image light, corrects opticalerrors associated with the image light, and presents the corrected imagelight to a user of the HMD 105. In various embodiments, the optics block118 includes one or more optical elements. Example optical elementsincluded in the optics block 118 include: an aperture, a Fresnel lens, aconvex lens, a concave lens, a filter, or any other suitable opticalelement that affects image light. Moreover, the optics block 118 mayinclude combinations of different optical elements. In some embodiments,one or more of the optical elements in the optics block 118 may have oneor more coatings, such as anti reflective coatings.

Magnification of the image light by the optics block 118 allows theelectronic display 115 to be physically smaller, weigh less, and consumeless power than larger displays. Additionally, magnification mayincrease a field of view of the content presented by the electronicdisplay 115. For example, the field of view of the displayed content issuch that the displayed content is presented using almost all (e.g., 110degrees diagonal), and in some cases all, of the user's field of view.In some embodiments, the amount of magnification may be adjusted byadding or removing optical elements.

The optics block 118 may be designed to correct one or more types ofoptical error. Examples of optical error include two dimensional opticalerrors, three dimensional optical errors, or some combination thereof.Two dimensional errors are optical aberrations that occur in twodimensions. Example types of two dimensional errors include: barreldistortion, pincushion distortion, longitudinal chromatic aberration,transverse chromatic aberration, or any other type of two-dimensionaloptical error. Three dimensional errors are optical errors that occur inthree dimensions. Example types of three dimensional errors includespherical aberration, comatic aberration, field curvature, astigmatism,or any other type of three-dimensional optical error. In someembodiments, content provided to the electronic display 115 for displayis pre-distorted, so the optics block 118 corrects the distortion whenit receives image light from the electronic display 115 generated basedon the content.

The IMU 130 is an electronic device that generates IMU data indicatingan estimated position of the HMD 105 relative to an initial position ofthe HMD 105 based on measurement signals received from one or more ofthe position sensors 125. A position sensor 125 generates one or moremeasurement signals in response to motion of the HMD 105. Examples ofposition sensors 125 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 130, or some combination thereof. The position sensors 125 may belocated external to the IMU 130, internal to the IMU 130, or somecombination thereof.

Based on the one or more measurement signals generated by the one ormore position sensors 125, the IMU 130 generates IMU data indicating anestimated position of the HMD 105 relative to an initial position of theHMD 105. For example, the position sensors 125 include multipleaccelerometers to measure translational motion (forward/back, up/down,left/right) and multiple gyroscopes to measure rotational motion (e.g.,pitch, yaw, roll). In some embodiments, the IMU 130 rapidly samples themeasurement signals from various position sensors 125 and calculates theestimated position of the HMD 105 from the sampled data. For example,the IMU 130 integrates the measurement signals received from one or moreaccelerometers over time to estimate a velocity vector and integratesthe velocity vector over time to determine an estimated position of areference point on the HMD 105. Alternatively, the IMU 130 provides thesampled measurement signals to the console 110, which determines the IMUdata. The reference point is a point that may be used to describe theposition of the HMD 105. While the reference point may generally bedefined as a point in space, in practice the reference point is definedas a point within the HMD 105 (e.g., a center of the IMU 130).

The IMU 130 receives one or more calibration parameters from the console110. As further discussed below, the one or more calibration parametersare used to maintain tracking of the HMD 105. Based on a receivedcalibration parameter, the IMU 130 may adjust one or more IMU parameters(e.g., sample rate). In some embodiments, certain calibration parameterscause the IMU 130 to update an initial position of the reference pointso the initial position of the reference point corresponds to a nextcalibrated position of the reference point. Updating the initialposition of the reference point as the next calibrated position of thereference point helps reduce accumulated error associated with thedetermined estimated position. The accumulated error, also referred toas drift error, causes the estimated position of the reference point to“drift” away from the actual position of the reference point over time.

The imaging device 135 captures one or more images of the local areasurrounding the HMD 105, with at least a set of the captured imagesincluding at least one SL pattern projected by the projector system 136.In various embodiments, the imaging device 135 may include one or morecameras, one or more video cameras, any other device capable ofcapturing images of the SL patterns projected by the projector system136, or some combination thereof. For example, the imaging device 135may comprising an RGB camera capable of capturing images in the visibleRGB spectrum, an IR camera capable of capturing images in the IRspectrum, or some combination thereof. In some embodiments, the imagingdevice 135 captures RGB data and IR concurrently, or may interleavecapture of RGB data and IR data. In some embodiments, captured RGB datamay be used to determine color or texture information of objects in thelocal area, in addition to depth and/or position information.

Additionally, the imaging device 135 may include one or more filters(e.g., for increasing signal to noise ratio). For example, the one ormore filters may comprise one or more bandpass filters based upon one ormore wavelength ranges associated with one or more of the projected SLpatterns. The imaging device 135 is configured to detect SL patterns ina field of view of the imaging device 135. In various embodiments, theimages captured by the imaging device 135 comprise image data that iscommunicated from the imaging device 135 to the console 110. The imagingdevice 135 receives one or more calibration parameters from the console110 to adjust one or more imaging parameters (e.g., focal length, focus,frame rate, ISO, sensor temperature, shutter speed, aperture, etc.) forcapturing images of the local area. In alternate embodiments, theimaging device 135 is separate from the HMD 105. For example, one ormore imaging devices may be configured to view the local area includingthe HMD 200 from different vantage points.

The tracking module 160 may track movements of the HMD 105 usingcaptured image data from one or more imaging devices 135. The trackingmodule 160 may also determine positions of the reference point of theHMD 105 using position information from the IMU data. Additionally, insome embodiments, the tracking module 160 may use portions of the IMUdata, the image data, or some combination thereof, to predict a futurelocation of the HMD 105. In some embodiments, the tracking module 160provides the estimated or predicted future position of the HMD 105 todetermine content to be displayed to the user through the electronicdisplay 115 (e.g., using the engine 165 of the console 110).

In some embodiments, the tracking module 160 calibrates the system 100using one or more calibration parameters and may adjust one or morecalibration parameters to reduce error in determination of the positionof the HMD 105. For example, the tracking module 160 may adjust thefocus of the imaging device 135 to obtain a more accurate position forobserved portion of projected SL patterns. Moreover, calibrationperformed by the tracking module 160 also accounts for informationreceived from the IMU 130 in the HMD 105. Additionally, if tracking ofthe HMD 105 is lost (e.g., the imaging device 135 loses line of sight ofsome portion of the projected SL patterns), the tracking module 160 mayre-calibrate some or all of the system 100. In some embodiments, thetracking module 160 may be implemented on the console 110 instead of theHMD 105.

The input interface 140 is a device that allows a user to send actionrequests to the console 110. An action request is a request to perform aparticular action. For example, an action request may be to start anapplication, to end an application, or to perform a particular actionwithin the application. The input interface 140 may include one or moreinput devices. Example input devices include: a keyboard, a mouse, agame controller, a joystick, a yoke, or any other suitable device forreceiving action requests and communicating the received action requeststo the console 110. An action request received by the input interface140 is communicated to the console 110, which performs an actioncorresponding to the action request. In some embodiments, the inputinterface 140 may also include an imaging device 135 that capturesimages of one or more light patterns projected by the projector system136 and provides the images to the console 110.

The input interface 140 may also include an IMU 130 that captures IMUdata indicating an estimated position of the input interface 140relative to an initial position of the VR interface 140 and provides theIMU data to the console 110. The IMU 130 receives one or morecalibration parameters from the console 110. As further discussed below,the one or more calibration parameters are used to maintain tracking ofthe input interface 140.

The input interface 140 may provide haptic feedback to the user inaccordance with instructions received from the console 110 in someembodiments. For example, haptic feedback is provided to the user whenan action request is received. As another example, the input interface140 provides haptic feedback to the user when the console 110communicates instructions to the input interface 140 causing the inputinterface 140 to generate haptic feedback when the console 110 performsan action.

The console 110 provides content to the HMD 105 for presentation to theuser in accordance with information received from one or more of: theimaging device 135, the HMD 105, and the input interface 140. In theexample shown in FIG. 1, the console 110 includes an application store145, a controller 150, a mapping module 155, and a virtual reality (VR)engine 165. Some embodiments of the console 110 have differentcomponents than those described in conjunction with FIG. 1. Similarly,the functions further described below may be distributed amongcomponents of the console 110 in different manners than described herein various embodiments.

The application store 145 stores one or more applications for executionby the console 110. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the HMD 105 or of the inputinterface 140. Examples of applications include: gaming applications,conferencing applications, video playback application, or other suitableapplications.

The controller 150 controls the projector system 136 by generating andproviding emission instructions to the projector system 136. In someembodiments, the controller generates emission instructions based upontypes of light sources and DOEs available in the projector system 136,one or more user settings (e.g., received via the input interface 140),one or more parameters received from the HMD 105 (e.g., the controller150 may adjust the emission instructions for the projector system 136based upon captured image information from the imaging device 135 of theHMD 105), or any combination thereof. In some embodiments, thecontroller 150 may also generate instructions for the imaging device 135on the HMD 105 for capturing images of the one or more objects in thelocal area illuminated by the projected SL patterns.

The mapping module 155 generates a virtual mapping of the local areabased on the images of projected SL patterns received from the HMD 105or from the input interface 140 (e.g., captured using the imaging device135). For example, the mapping module 155 determines locations ofportions of the SL patterns projected in the local area relative to theprojector system 136 and to the HMD 105. For example, the mapping module155 uses image data (e.g., images of portions of the local area) fromthe HMD 105 to calculate distances from portions of the projected SLpatterns to the HMD 105. From the information received from the HMD 105and the SL pattern information from the projector system 136, themapping module 155 generates a virtual mapping by associating portionsof the projected SL patterns to different locations in a virtual spacethat overlays the local area.

The VR engine 165 executes applications within the system 100 andreceives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof, ofthe HMD 105 from the tracking module 160. Based on the receivedinformation, the VR engine 165 determines content to provide to the HMD105 for presentation to the user. Content may include video information,one or more images, virtual objects, audio information, or somecombination thereof. For example, if the received information indicatesthat the user has looked to the left, the VR engine 165 generatescontent for the HMD 105 that mirrors the user's movement in a virtualenvironment. Additionally, the VR engine 165 performs an action withinan application executing on the console 110 in response to an actionrequest received from the input interface 140 and provides feedback tothe user that the action was performed. The provided feedback may bevisual or audible feedback via the HMD 105 or haptic feedback via theinput interface 140.

FIG. 2 is a wire diagram of a HMD 200, in accordance with an embodiment.The HMD 200 is an embodiment of the HMD 105, and includes a front rigidbody 205 and a band 210. The front rigid body 205 includes one or moreelectronic display elements of the electronic display 115 (not shown),the IMU 130, the one or more position sensors 125, and the imagingdevice 135. In some embodiments, the imaging device 135 comprises twodifferent cameras, wherein separate images produced by the two camerasare used to determine distance from the HMD 200 to a portion of the SLpatterns projected by projector system 136 imaged by both cameras. Insome embodiments, the two cameras may be wide angle cameras withoverlapping fields of view. In other embodiments, the imaging device 135may comprise a single camera.

Tileable Light Patterns

In some embodiments, the projector system projections multiple SLpatterns into the local area, each associated with a tileable boundary.Each projected SL pattern may share an edge of the tileable boundarywith that of another projected SL pattern, minimizing a gap between theprojected SL patterns. In addition, because each SL pattern is confinedwithin its respective tileable boundary, an amount of overlap between SLpatterns is minimized. As such, the SL patterns may cover a wide FOV ofthe local area.

FIG. 3 is a block diagram of an embodiment of a projector system 300including multiple projection assemblies 310A, 310B, and 310C. Theprojector system 136 is an embodiment of the projector system 300. Inthe example shown by FIG. 3, the projector system 300 includes a sourceassembly 305 and projection assemblies 310A, 310B, and 310C. The sourceassembly 305 is a coherent light source configured to emit coherentbeams of light 315 (e.g., beams of light 315A, 315B, and 315C) directedto the projection assemblies 310A through 310C. Examples of the sourceassembly 305 include a laser diode, a vertical cavity surface emittinglaser, a tunable laser, or another light source that emits coherentlight. In various embodiments, the source assembly 305 emits light inthe IR band; however, in other embodiments, the source assembly 305emits light in the visible band, in the UV band, or in any othersuitable band. In some embodiments, the beams of light 315 may berelatively collimated. However, in some embodiments, the source assembly305 emits beams of light 315 that are not collimated. For example, thelight emitted by the source assembly 305 may be diverging or converging.Hence, in some embodiments, the source assembly 305 also includes acollimator that collimates light from a light source into the beam oflight 315. In some embodiments, the different beams of light 315A, 315B,315C output by the source assembly 305 are in the same ranges ofwavelengths. Alternatively, different beams of light 315A, 315B, 315Coutput by the source assembly 305 are in different ranges ofwavelengths. In addition, although FIG. 3 illustrate the beams of light315A, 315B, and 315C as single beams of light, it is understood that inother embodiments, each beam of light 315A, 315B, 315C may comprisemultiple beams of light.

The projection assemblies 310A, 310B, and 310C each receives arespective beam of light 315A, 315B, or 315C emitted from the sourceassembly 305 and outputs a respective SL pattern. In some embodiments,each of the projection assemblies 310A, 310B, and 310C includes adifferent ADOE, so each projection assembly 310A, 310B, and 310C outputsa different SL pattern 342A, 342B, and 342C, respectively. In alternateembodiments, the ADOEs in each of the projection assemblies 310A, 310B,and 310C is the same, but each projection assembly 310A, 310B, 310C isilluminated using a different range of wavelengths. For example, beam oflight 315A is a particular range of wavelengths, beam of light 315B is adifferent range of wavelengths, and beam of light 315C is another rangeof wavelengths. Accordingly, the SL patterns 342A, 342B, and 342C mayhave the same spatial configuration, but have different reflectancetypes; thus, each of the SL patterns 342A, 342B, 342C, still has aunique location configuration.

Using multiple projection assemblies 310A, 3108, and 310C allows for theprojector system 300 to project SL patterns over a wider FOV, whileavoiding the problems of using a single DOE to project a SL pattern overa wide FOV (e.g., large zero-order values relative to other diffractionorders, a large number of diffraction orders, etc.). For example, eachof the SL patterns may have an FOV of approximately 60 degrees, allowingthe projector system 300 to project SL patterns over a 180 degree viewonto the local area. In addition, the use of multiple projectionassemblies may allow for a greater density of light beams for eachprojected SL pattern, potentially allowing a more accurate mapping ofthe local area.

As discussed above, each of the projection assemblies 310A, 310B, and310C comprises an ADOE configured to restrict the SL pattern projectedby the respective projection assembly to within a tileable boundary. Assuch, projected SL patterns of adjacent projection assemblies 310Athrough 310C can be configured to tessellate with each other over theFOV of the projector system 300, with minimal overlap or gaps betweenadjacent SL patterns.

For example, in some embodiments, each ADOE corresponds to a rectangularboundary. In some embodiments, the formula used to create theappropriate masked pattern for design into the ADOE and preventingprojection of specific light that would otherwise lie outside a tileableboundary is given below for generating a SL pattern projection with arectangular FOV of size θ_(h) degree×θ_(v) degree:

$\begin{matrix}{{M\left( {i_{h},i_{v}} \right)} = \left\{ \begin{matrix}{1,} & {\;{\begin{matrix}{{{\left( {i_{h} - \left( {n_{h} + 1} \right)} \right){\sin\left( \alpha_{h} \right)}}} \leq {\sin\mspace{11mu}\left( \frac{\theta_{h}}{2} \right)\mspace{11mu}{AND}}} \\{{{\left( {i_{v} - \left( {n_{v} + 1} \right)} \right){\sin\left( \alpha_{v} \right)}}} \leq {\sin\mspace{11mu}\left( \frac{\theta_{v}}{2} \right)}}\end{matrix},}\;} \\{0,} & {otherwise}\end{matrix} \right.} & (1)\end{matrix}$where θ_(h) is an angular field of view in a horizontal direction, θ_(v)is an angular field of view in a vertical direction, 1≤i_(h)≤2n_(h)+1,1≤i_(v)≤2n_(v)+1, (2n_(h)+1)×(2n_(v)+1) is a size of the original designpattern bitmap, i_(h) is an integer that represents a horizontalcoordinate in the design pattern bitmap, i_(v) is an integer thatrepresents a vertical coordinate in the design pattern bitmap, n_(h) isan integer that represents an index in horizontal direction of thedesign pattern bitmap, n_(v) is an integer that represents an index invertical direction of the design pattern bitmap, M(i_(h),i_(v)) is anormalized intensity of an emitted light beam in a position in thedesign pattern bitmap after being diffracted by the ADOE, α_(h) is adiffraction order separation in horizontal direction, and α_(v) is adiffraction order separation in vertical direction. As used herein,horizontal and vertical directions may be based upon a plane associatedwith the ADOE (e.g., a plane parallel orthogonal to a direction of thezero order diffraction of the ADOE). In some embodiments, the ADOEs ofeach of the projection assembly 310A, 310B, and 310C may be associatedwith the same plane. In other embodiments, each of the ADOEs may beassociated with different planes.

Although FIG. 3 illustrates each of the projection assemblies 310A,310B, and 310C as separate components, in some embodiments, two or moreADOE structures of the projection assemblies 310A, 310B, and 310C areprinted on the same substrate. Functionally, the ADOEs printed on thesame substrate are equivalent to multiple physically separated ADOEs.However, physically, ADOEs printed on the same substrate representing asingle part may simplify manufacturability. In addition, although FIG. 3illustrates a single source assembly 305 projecting multiple beams oflight 315A-C towards multiple projection assemblies 310A-C, it isunderstood that in other embodiments, the projector system 300 maycomprise multiple source assemblies 305 (e.g., a different sourceassembly 305 for each of the projection assemblies 310A-C).

In some embodiments, the SL patterns projected by the projector systemare tiled or tessellated such that the projected SL patternscollectively define a curved or non-planar surface. For example, insteadof projecting SL patterns that tessellate along a particular plane, eachof the projected SL patterns may define a different plane.

FIG. 4 illustrates a diagram of a projector system projecting tileablelight patterns, in accordance with some embodiments. As discussed above,each projected light pattern projected by the projector system 400 isdefined by a boundary having a particular shape. For example, eachprojected light pattern may have a boundary having a substantiallyrectangular shape.

The projector system 400 configured to project at least three differentstructured light patterns (SL patterns 405A, 405B, and 405C). Thedifferent projected SL patterns are tiled such that they collectivelydefine a curved or non-planar surface. For example, the first, second,and third SL patterns 405A, 405B, and 405C are arranged such that theplanes defined by each SL pattern are offset by an angle (e.g., anglesθ1 and 02) relative to that of the previous SL pattern. In addition,each of the SL patterns may define a substantially equal FOV.

In some embodiments, projecting different SL patterns such that theydefine a non-planar surface may allow for the projector system to betterproject SL patterns over objects in the local area surrounding theprojector system. For example, the local area may correspond to a roomthat the projector 400 is located in, wherein different objects in thelocal area (e.g., walls, ceilings, furniture) may be orienteddifferently relative to the projector system. By projecting SL patternsdefined by different planes, the light beams of the SL pattern may beprojected more uniformly over the objects of the local area compared toif the SL patterns are all defined by the same plane (e.g., the verticalplane defined by the first SL pattern 405A as illustrated in FIG. 4),allowing for positions or depths of the objects in the local area to bemore easily and accurately calculated.

FIG. 5 illustrates another diagram of a projector projecting tileablenon-coplanar SL patterns, in accordance with some embodiments. In someembodiments, the projector system 500 projects a plurality of differentSL patterns defined by different planes over a 360° area around theprojector. For example, as illustrated in FIG. 5, the projector system500 is configured to project 6 different SL patterns. Each SL pattern isconfigured to cover a FOV of approximately 90° from the projector, andare arranged such that the planes defining the SL patterns collectivelyform a cube around the projector. The six projected SL patterns maycomprise four SL patterns 505A through 505D corresponding to fourvertical faces of the cube, as well as two additional SL patterns 510and 515 corresponding respectively to a top face and a bottom face ofthe cube.

In some embodiments, the projector system 500 may project fewer or moreSL patterns than that illustrated in FIG. 5, such that SL patterns willnot be projected onto certain portions of the local area. For example,the projector system 500 may be configured to project SL patternscorresponding to the side faces 505A-505D and top face 510, but not thebottom face 515 of the cube, due to a lack of any objects of interestdirectly below the projector. In some embodiments, the projector system500 may project a different number of SL patterns and/or SL patternswith different FOVs. For example, in an embodiment, instead ofprojecting four rectangular SL patterns corresponding to vertical facessurrounding the projector, each with an FOV of approximately 90°, theprojector may project three rectangular SL patterns each with an FOV ofapproximately 120°. In some embodiments, each of the SL patternsillustrated in FIG. 5 may comprise a plurality of smaller coplanar SLpatterns.

Although FIGS. 4 and 5 illustrate rectangular SL patterns, it isunderstood that in other embodiments, projected SL patterns may bedefined by boundaries having different shapes. For example, in aparticular embodiment, each SL pattern may have a boundary shapecorresponding to a pentagon, and wherein the plurality of SL patternsare tiled based upon a shape of a dodecahedron.

In some embodiments, each projected SL pattern may be associated with acurved surface. For example, FIG. 6 illustrates a projector 600projecting a SL pattern associated with a curved surface. The projectedSL pattern may be defined by a boundary 605. The boundary 605 isassociated with a plane 610, which corresponds to a plane orthogonal toa direction of the zero order diffraction of an ADOE associated with theSL pattern. In addition, the boundary 605 may be associated with acurved surface 615, which corresponds to a projection of the plane 610.

In some embodiments, the curved surface 615 corresponds to a portion ofa surface of a sphere centered at the location of the projector 600. Assuch, the plurality of SL patterns projected from a projector system maycollectively define a sphere. For example, the projector system 500 ofFIG. 5 may project 6 SL patterns, each corresponding to a curvedsurface, wherein the six curved surfaces corresponding to the six SLpatterns collectively define a substantially spherical surface.

FIG. 7 illustrates a flowchart of a process for projecting SL patternsin accordance with some embodiments. The process of FIG. 7 may beperformed by an HMD system (e.g., the HMD system 100). Other entities(e.g., a HMD) may perform some or all of the steps of the process inother embodiments. Likewise, embodiments may include different and/oradditional steps, or perform the steps in different orders.

The HMD system emits 702 (e.g., via the projector system 136) a first SLpattern having a first tileable boundary onto a first portion of one ormore objects in a local area surrounding at least a portion of the DCA.In some embodiments, the boundary of the first SL pattern issubstantially rectangular in shape, and covers a FOV of approximately90°.

The HMD system emits 704 a second SL pattern having a second tileableboundary onto a second portion of the one or more objects of the localarea, wherein the first and second tileable boundaries of the first andsecond SL patterns share at least one edge and collectively define anon-planar surface. In some embodiments, the second tileable boundary ofthe second SL pattern is substantially rectangular in shape, and isnon-overlapping with the first tileable boundary. The second SL patternmay cover an FOV of approximately 90°, and be oriented approximately 90°relative to a plane of the first SL pattern.

The HMD system captures 706 via an imaging device (e.g., the imagingdevice 135) one or more images of the one or more objects in the localarea that are illuminated by the first and second SL patterns. In someembodiments, the imaging device is located on an HMD separate from theprojector system. In some embodiments, an image of the one or moreimages may capture a portion of both the first and second SL patterns(e.g., at an area around the shared edge of the first and second SLpatterns). In embodiments where the first and second SL patterns areassociated with different wavelength ranges, each image of the one ormore images may only capture portions of either the first or second SLpatterns, but not both.

The HMD system determines 708 depth information for the one or moreobjects in the local area using the one or more images. For example, theHMD system may use the positions of the first and second SL patternswithin the one or more captured images to determine a position of an HMDon which the imaging device is mounted relative to a base stationcontaining the projector system. In some embodiments, the HMD systemuses the positions of the SL patterns within the captured images todetermine a depth of one or more objects onto which the SL patterns areprojected from the imaging device.

The HMD system provides 710 the determined depth information to aconsole or a HMD of the HMD system. In some embodiments, the console orthe HMD may use the determined depth information to generate VR or ARcontent for display to a user. For example, the console or the HMD maygenerate one or more virtual objects to be displayed to a user, basedupon the detected positions of one or more objects in the local area.

Additional Configuration Information

The foregoing description of the embodiments of the disclosure have beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A head-mounted display (HMD), comprising: adetector assembly comprising at least one camera configured to captureone or more images of the one or more objects in a local areailluminated by a tiled structured light (SL) pattern generated by aprojector assembly, the tiled SL pattern comprising: a first SL patternprojected over a first field of view (FOV) corresponding to a firsttileable boundary, and a second SL pattern projected over a second FOVcorresponding to a second tileable boundary, wherein the first andsecond tileable boundaries share at least one edge and collectivelydefine a non-planar surface; and a processor comprising a controllerconfigured to determine a location of the HMD using the one or morecaptured images.
 2. The HMD of claim 1, wherein the first SL pattern isprojected through a first diffractive optical element (DOE) comprising apattern mask that prevents projection of light that would otherwise bediffracted into an area outside the first tileable boundary.
 3. The HMDof claim 1, wherein the first and second tileable boundariescollectively define a curved surface.
 4. The HMD of claim 1, wherein thefirst SL pattern has an FOV of 90°.
 5. The HMD of claim 1, wherein thefirst tileable boundary of the first SL pattern defines a first plane,and the second tileable boundary of the second SL pattern defines asecond plane, wherein the first and second planes are oriented 90°relative to each other.
 6. The HMD of claim 1, wherein the firsttileable boundary is rectangular.
 7. The HMD of claim 1, wherein thefirst SL pattern is projected using a first range of wavelengths, andthe second SL pattern is projected using a second range of wavelengthsdifferent from the first range.
 8. The HMD of claim 1, wherein thecontroller is further configured to provide the determined depthinformation to a console, and the console is configured to generatecontent for presentation on an electronic display of the HMD based uponthe depth information.
 9. A depth camera assembly, comprising: aprojector assembly configured to emit a structured light (SL) patternonto one or more objects in a local area, comprising: a plurality oflight emitters that emit a plurality of light beams; a first diffractiveoptical element (DOE) configured to diffract emitted light beams from afirst subset of the plurality of light emitters to form a first SLpattern having a first field of view (FOV) corresponding to a firsttileable boundary, a second DOE configured to diffract emitted lightbeams from a second subset of the plurality of light emitters to form asecond SL pattern having a second FOV corresponding to a second tileableboundary, wherein the first and second tileable boundaries share atleast one edge and collectively define a non-planar surface; a detectorassembly comprising at least one camera configured to capture one ormore images of the one or more objects in the local area illuminated bythe tiled SL pattern; and a processor comprising a controller configuredto determine a location of the HMD using the one or more capturedimages.
 10. The depth camera assembly of claim 9, wherein the first DOEcomprises a pattern mask that prevents projection of light that wouldotherwise be diffracted into an area outside the first tileableboundary.
 11. The depth camera assembly of claim 9, wherein the firstand second tileable boundaries collectively define a curved surface. 12.The depth camera assembly of claim 9, wherein the first SL pattern hasan FOV of 90°.
 13. The depth camera assembly of claim 9, wherein thefirst tileable boundary of the first SL pattern defines a first plane,and the second tileable boundary of the second SL pattern defines asecond plane, wherein the first and second planes are oriented 90°relative to each other.
 14. The depth camera assembly of claim 9,wherein the first tileable boundary is rectangular.
 15. The depth cameraassembly of claim 9, wherein the first SL pattern is projected using afirst range of wavelengths, and the second SL pattern is projected usinga second range of wavelengths different from the first range.
 16. Thedepth camera assembly of claim 9, wherein the detector assembly is partof a head-mounted display (HMD).
 17. The depth camera assembly of claim9, wherein the controller is further configured to provide thedetermined depth information to a console, and the console is configuredto generate content for presentation on an electronic display of ahead-mounted display (HMD) based upon the depth information.